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STANDARDS  CONFORMANCE  TESTING: ISSUES  AND  ACTIVITIES 


INTRODUCTION 

l 

Purpose.  To  provide  guidance  when  developing  a conformance 
testing  program  for  a Federal  Information  Processing  Standard 
(FIPS) . 

Scope.  Anyone  interested  in  developing  a conformance  testing 
program  may  find  this  document  provides  a starting  point  from 
which  to  depart. 

Background . 

Technically  sound  national  and  international  standards  are  needed 
to  preserve  open  competition  in  international  markets  and  to 
support  increased  productivity  and  delivery  of  services  at 
reduced  cost.  Through  standards,  users  are  provided  with  off- 
the-shelf,  compatible  hardware,  software  and  communications 
products  for  computer  and  related  telecommunications  systems. 
Validation  of  these  computer  products  claiming  conformance  with 
standards  further  reduces  risks  and  uncertainties  to  vendors  and 
users?  uniform  conformance  testing  procedures  should  be  employed 
to  perform  this  validation. 

Conformance  testing  constitutes  the  examination  of  a candidate 
product  or  service  for  the  existence  of  specific  characteristics 
required  by  a standard.  The  Institute  for  Computer  Sciences  and 
Technology  (ICST)  at  the  National  Bureau  of  Standards  (NBS) 
participates  in  development  of  U.S.  Government-wide  standards  for 
computer  software,  hardware,  data  management,  networks  and 
security.  ICST  works  through  voluntary  industry  standards 
organizations  to  develop  standards  that  will  meet  the  needs  of 
Government  users  and  be  implemented  in  off-the-shelf  commercial 
products.  Standards  that  promise  sizable  benefits  to  the 
Government  are  issued  as  Federal  Information  Processing  Standards 
(FIPS) . These  FIPS  and  the  specifications  they  adopt  are 
implemented  into  computer  products  for  commercial  use.  Through 
past  experience  in  research  and  testing,  ICST  sees  a need  for 
expansion  of  its  efforts  in  structuring  conformance  testing  to 
these  FIPS,  and  is  in  the  process  of  formulating  a program. 

ICST  is  developing  an  initial  structure  and  approach  for  uniform 
conformance  testing  programs  Government-wide  for  FIPS.  This 
paper  presents  a conceptual  framework  which  is  modeled  after  such 
established  international  programs  as  the  National  Voluntary 
Laboratory  Accreditation  Program  (NVLAP)  and  language  compiler 
validation  program;  as  well  as  the  international  efforts  in  the 
ISO  graphics  committee.  This  framework  outlines  a progression  of 
steps  needed  to  develop  a conformance  testing  program  for  any 
given  standard.  (Although  this  document  does  not  particularly 
address  verification  or  acceptance  testing,  some  of  these  steps 
may  also  apply  for  these  programs.) 
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One  of  the  issues  needing  to  be  addressed  separately  is  the 
technical  approach  to  program  and  test  suite  development. 
Another  area  of  concern  is  available  resources.  Staff  and 
equipment  required  for  development,  continual  maintenance  and 
updating,  and  all  documentation  are  just  a few  considerations 
when  developing  a conformance  testing  program.  An  additional 
critical  element  impacting  effective  use  of  the  given  resources 
is  timings  when  does  the  user  community  need  these  conformance 
tests,  and  how  long  will  it  take  to  provide  the  tests  using  the 
current  level  of  resources? 

Part  of  each  standard  conformance  testing  program  development 
will  be  recognition  of  the  unique  characteristics  of.  each 
standard,  such  as:  stability  for  conformance  test  development; 

level  to  which  evaluation  can  be  automated;  "degree  of 
completeness"  of  tests  (when  is  enough,  enough?) ; test  expansion; 
interpretation  of  the  standard  for  test  development;  (should 
tests  be  developed  before,  after  or  concurrent  with  standard 
development) ; criteria  (should  all  tests  have  pass/fail  results) ; 
human  intervention  allowed  during  testing;  and  the  potential 
implication  of  excluding  portions  of  the  standard  for  commercial 
use  since  only  selected  portions  of  the  standard  may  be  tested. 

The  issues  and  procedures  identified  in  this  paper  are  among 
those  that  will  be  considered  in  developing  the  overall  ICST 
Conformance  Testing  Program.  The  ICST  conformance  program  and 
individual,  conformance  tests  for  any  given  standard  will  evolve 
as  more  experience  is  gained. 

Document  Organization. 

The  remainder  of  this  document  is  divided  into  two  parts;  Status 
of  ICST 8 s Conformance  Testing  Programs,  which  provides  a brief 
summary  of  conformance  testing  programs  already  under  way  at 
ICST;  and  Conceptual  Framework  for  Developing  Conformance  Testing 
Programs . This  second  part  is  an  outlined  process  offering  one 
approach  for  developing  a conformance  testing  program  for  any 
given  standard.  Variations  to  this  approach  are  assumed  since 
testing  requirements  and  programs  will  be  based  on  the  maturity 
and  complexity  of  the  standard,  available  resources,  and  public 
demand.  It  is  broken  down  into  the  following  sections; 

1.  Program  Initiation 

2 . Conformance  Testing  Program  Development 

3 . Testing  Methods 

4.  Certification  and  Reporting  Considerations 

5.  Testing  Administration 

6.  Testing  Laboratories 

7.  Mutual  Recognition  of  Validation/Certification 

8.  References 


APPENDIX 


Terms  and  Definitions 
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APPENDIX  B.  Sample  Application  for  Testing  Laboratory 
APPENDIX  C.  Model  Certificate  of  Accreditation 

Terms. 

Appendix  A provides  terms  and  their  definitions  used  within  the 
body  of  this  report  or  commonly  used  when  discussing  conformance 
testing.  They  are  presented  to  provide  a common  understanding 
from  which  to  depart. 
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STATUS  OF  ICST  CONFORMANCE  TESTING  PROGRAMS 


The  following  paragraphs  provide  a glimpse  of  initiatives  in 
conformance  test  development  at  ICST.  Various  approaches  were 
employed  for  a common  goal:  testing  conformance  to  a standard. 
Some  programs  are  more  complex  than  others  due  to  the  varying 
requirements  and  nature  of  the  standards  themselves.  Development 
has  been  completed  on  only  a few  of  the  programs. 

Compiler  Validation.  ICST  provides  compiler  validation  for  the 
following  FIPS  languages:  COBOL  ( FIPSPUB2 1-2 ) , FORTRAN 
(FIPSPUB69-1) . PASCAL  (FIPSPUB109) , Ada  (FIPSPUB119) , and  BASIC 
(FIPSPUB68-1)  . There  is  a fee  associated  with  each  language 
compiler  test  to  recover  costs  since  it  has  been  a multimillion 
dollar  investment  to  date.  An  entire  ICST  certification  program 
is  in  place,  which  includes  complete  validating  and  operating 
procedures,  and  publishing  a certified  list  of  those  compilers 
currently  under  certification.  The  quantity  of  tests  in  the  test 
suite  will  vary;  for  example,  PASCAL  currently  has  approximately 
800  tests  associated  with  it,  Ada  has  approximately  2600.  ICST 
cooperates  with  other  testing  and  certification  authorities  in 
the  United  States  (e.g.,  Department  of  Defense  Ada  Joint  Program 
Office) , as  well  as  internationally  (France,  the  UK  and  Germany) . 
The  Federal  Information  Resources  Management  Regulations2 
requires  all  compilers  sold  to  the  U.S.  Government  must  show 
conformance  to  the  appropriate  language  FIPS. 

Document  interchange.  The  International  Standard  Generalized 
Markup  Language  (SGML)  (proposed  as  a FIPS) , requires  "parsing" 
software  to  test  the  conformance  of  document  type  definitions  and 
documents  to  the  standard.  SGML  tools  that  parse  SGML  documents 
also  need  to  be  checked  for  completeness  and  ICST  has  developed  a 
validation  suite  of  test  cases  to  do  this.  ICST  has  developed 
and  made  publicly  available  an  SGML  reference  parser,  an  SGML 
validation  suite,  and  supporting  documentation.  This  software  is 
available  for  a nominal  fee  from  NTIS  (703/487-4650)  . No  formal 
certification  program  is  in  place,  and  ICST  does  not  endorse  or 
certify  any  product  which  has  successfully  been  parsed  using  this 
software . 

Operating  Systems.  The  standard  for  Portable  Operating  System 
Interface  for  Computer  Environments  (POSIX)  has  been  proposed  as 
a FIPS.  A reference  implementation  test  suite  is  also  being 
developed  to  test  conformance  to  the  FIPS.  This  implementation 
is  based  on  the  draft  document  being  developed  in  the  P1003.3 


FIPSPUBS  are  available  through  U.S.  Department  of 
Commerce,  National  Technical  Information  Service, 
Springfield,  VA  22161?  703/487-4650. 

FIRMR  201-8.109 
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Committee  sponsored  by  IEEE.  A number  of  companies  are 
participating  in  this  implementation.  When  completed,  the  source 
code  for  the  test  suite  will  be  placed  in  the  public  domain. 
ICST  is  investigating  the  use  of  third  party  testing  services, 
and  plans  to  maintain  and  update  the  test  suite  as  the  standard 
evolves.  The  initial  release  of  the  subset  is  scheduled  CY1988. 

Graphics.  An  effort  for  the  Graphics  Kernel  System  (GKS) 
(FXPSPUB120)  validation  began  in  1982  and  has  been  funded  by  the 
Commission  of  the  European  Communities  (CEC  or  EC)  . ICST  has 
been  an  active  participant  along  with  Germany,  the  United  Kingdom 
and  France.  The  falsification  testing  methodology  has  been 
employed,  tests  have  been  developed  for  five  separate  interfaces 
(application,  data  structure,  error  data  consistency,  metafile 
device,  and  operator  {user}),  and  ICST  has  provided  35  beta  test 
sites  with  copies  for  comment.  ICST  has  been  welcomed  into  CEC  as 
a potential  test  site  for  future  testing  of  both  GKS  and  CGM 
(Computer  Graphics  Metafile)  (FXPSPUB128) . A one  year  trial  test 
service  is  scheduled  to  begin  in  1988.  Work  is  also  progressing 
toward  developing  test  suites  for  CGM  and  PHIGS,  the  Programmer's 
Hierarchical  Interactive  Graphics  Standard  (anticipated  as  a FIPS 
CY1988 ) . 

Data  Management.  The  Information  Resource  Dictionary  System 
(IRDS)  has  been  the  forerunner  in  the  data  management  standards 
for  initiating  a conformance  testing  program.  Currently 
completing  the  final  stages  of  becoming  a national  standard,  it 
will  also  be  a FIPS.  ICST  has  developed  a prototype 
implementation  of  a subset  of  the  IRDS  command  language;  future 
direction  includes  a complete  reference  implementation.  The 
prototype  has  been  made  available  for  vendors  who  are  willing  to 
provide  feedback  on  their  results.  Industry  is  also  cooperating 
in  the  development  of  tests.  Database  Language  SQL  (FIPSPUB127 ) , 
comprised  of  a schema  definition  language  and  a module  language 
which  includes  structured  query  language  statements,  has  a 
validation  test  methodology  and  test  suite  also  under 
development. 

Raster  Compression.  In  support  of  the  Department  of  Defense,  ICST 
is  currently  prototyping  the  compression  and  decompression  of 
engineering  drawings,  for  storage  and  transmittal,  using  the 
CCITT , Group  4 Raster  Compression  standard  (FED-STD  1064  & 1065) . 
The  testing  is  demonstrating  8 1/2 "xll"  size  drawings;  ICST 
anticipates  that  a formal  compression  conformance  testing  program 
will  be  established  in  CY1988. 

Networking. 

FDDI.  The  Fiber  Distributed  Data  Interface  (FDDI)  is  a token  ring 
network  standard  which  allows  up  to  1000  stations  in  a network. 
ICST  has  established  a Fiber  Optic  Network  Laboratory,  initially 
focusing  on  FDDI.  A station  test  set  and  procedures  were 
developed  and  a neutral,  instrumented  testbed  for  multi-vendor 
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fiber  LAN  integration  testing  is  being  established.  Applying 
these  facilities  and  documentation,  ICST  cooperates  with  industry 
for  use  and  participation  in  developing  commercial  test 
equipment . 

X.  25 . ICST  developed  a testing  methodology  for  the  International 
Telegraph  and  Telephone  Consultative  Committee  (CCITT) 
Recommendation  X.25  in  1983.  The  Department  of  Defense,  Navy  and 
Army  have  adopted  the  method,  and  the  ICST  test  method  is  being 
used  as  a base  to  establish  a national  standard  on  the  testing  of 
X.25.  ISO/TC97/SC6  has  established  a project  (ISO  8882)  to  test 
X.25  terminals.  Additional  efforts  for  a conformance  testing 
framework  are  underway  in  the  international  standards  arena. 
FIPS  PUB  122  provides  the  conformance  tests  for  FIPS  PUB 
100/FEDSTD  1041. 

ISDN.  ICST  has  solicited  support  from  other  government  agencies 
and  industry,  and  is  developing  testing  specifications  and 
coordinating  efforts  with  the  Open  System  Interconnection  (OSI) 
testing  initiatives.  Next  steps  include  developing  a standard 
test  report  format,  implementing  the  test  specifications,  issuing 
a FIPS  on  the  Integrated  Services  Digital  Network  (ISDN) 
conformance  testing,  developing  a plan  on  ISDN  laboratory 
accreditation,  and  attracting  national  and  international  support 
of  the  testing  program. 

GOSIP.  ICST  has  developed  a FIPS  for  the  Government  Open  System 
Interconnection  Profile  (GOSIP)  specification.  GOSIP  will  enable 
Federal  agencies  to  acquire  and  use  OSI  protocols  rather  than 
vendor-specific  or  proprietary  protocols.  GOSIP  is  expected  to 
be  promulgated  as  a FIPS  in  early  CY1988.  ICST  has  already 
developed  a prototype  testing  network  called  OSINET,  a global 
network  supporting  25  industry  and  government  organizations. 
This  cooperative  testing  among  vendors  through  demonstration 
participation,  the  availability  of  such  an  OSINET  operation,  and 
research  and  development  activities  at  ICST  are  all  initiatives 
contributing  to  the  testing  program  development.  Some  of  the 
tests  have  been  developed  by  ICST;  the  Corporation  for  Open 
Systems  (COS)  is  also  developing  tests.  Organizations  such  as 
the  Industrial  Technology  Institute  (ITI)  already  offer  testing. 
Overall,  tests  and  testing  programs  will  be  evaluated  by  ICST  for 
their  suitability  and  sufficiency  in  meeting  Federal 
requirements . 

Storage  Media.  NBS  offers  for  sale  various  standard  reference 
materials  (SRM)  for  evaluating  the  performance  of  magnetic 
storage  media  and  systems,  and  for  maintaining  control  over  their 
production.  Each  SRM  is  individually  calibrated  and  certified. 
Purchase  orders  for  all  SRM's  should  be  addressed  to:  Office  of 
Standard  Reference  Materials,  Room  B311  Chemistry  Building, 
National  Bureau  of  Standards,  Gaithersburg,  MD  20899;  telephone 
3 Ol/975-OSRM  {6776};  FTS  879-OSRM;  Telex  TRT197674NBS  UT. 
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Security.  DES  Validation.  ICST  has  had  in  place  a Data 
Encryption  Standard  (DES)  Validation  Program  since  1977  which 
validates  DES  implementations  for  conformance  to  the  standard 
{ FIPSPUB46 } . In  1986  a Message  Authentication  Code  (MAC) 
Validation  System  was  established,  based  on  FIPS  PUB  113 # 
Computer  Data  Authentication.  For  both  of  these  services,  ICST 
provides  Certificates  of  Validation  for  devices  which 
successfully  pass  the  tests.  To  date,  ICST  has  validated  more 
than  10  vendor  equipments  for  MAC  (two  of  which  were  trans- 
Atlantic  (UK,  Denmark}),  and  more  than  25  for  DES.  A Key 
Management  Validation  System  which  will  support  the  testing  of 
ANSI  X9.17,  Financial  Institution  Key  Management,  is  anticipated 
to  be  in  place  in  CY1988. 
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CONCEPTUAL  FRAMEWORK  FOR  DEVELOPING 
CONFORMANCE  TESTING  PROGRAMS 


1 . PROGRAM  INITIATION 

1.1.  Standards  Identification. 

Specific  FIPS  and  other  standards  being  prepared  for  FIPS 
publication  are  currently  under  study  as  candidates  for 
conformance  testing.  Considering  the  following  will  help 
identify  those  FIPS  requiring  a conformance  testing  program: 

1.1.1.  Is  the  standard  straightforward  so  that 
conformance  to  it  would  be  obvious  and  tests 
unnecessary,  e.g.,  state  and  country  codes. 

LL2.  What  are  the  needs  and  scope  of  the  user 
population. 

1.1.3.  What  is  the  nature  and  content  of  other  relevant 
conformance  testing  programs ...  are  there  already 
sufficient  programs  in  place  to  test  the  standard. 

1.1.4.  What  is  the  current  or  perceived  future 
importance  of  the  FIPS  to  commerce,  or  to  consumer 
well-being. 

1.1.5.  What  is  the  economic  and  technical  feasibility 
of  accrediting  testing  laboratories  for  the  test 
methods,  types  of  test  methods,  products,  or  services 
requested. 

2.  CONFORMANCE  TESTING  PROGRAM  DEVELOPMENT 

2.1.  Guide  for  Developing  a Testing  Program. 

The  following  issues  need  to  be  addressed  in  developing  a 
new  testing  program;  guidelines  and  recommendations  to  solve 
these  issues  follow  in  the  remainder  of  the  text. 

2.1.1.  If  multiple  test  methods  are  currently 
available,  establish  assessment  techniques  and 
priorities  to  decide  which  test  method (s)  should  be 
offered  for  testing. 

2.1.2.  Decide  what  the  units  of  evaluation  should  be, 
i.e.,  should  the  test  methods  be  offered  individually 
or  should  they  be  logically  grouped;  should  all  test 
methods  of  a standard  specification  be  offered  as  a 
group . 

2.1.3.  Identify  the  critical  elements  (environment, 
test  equipment  and  apparatus,  procedures) , for  each 
test  method,  e.g.,  quality  assurance  checks. 
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2.1.4.  Establish  assessment  techniques  and  priorities 
based  on  the  complexity  and  importance  of  each  test 
method,  e.g„,  should  the  assessor  carry  any 
instruments;  should  any  test  methods  be  demonstrated 
during  assessment. 

2.1.5.  Determine  the  applicability  of  field  testing 
important  test  methods  and  the  means  for  implementing 
new  or  available  field  testing  programs. 

2.1.6.  Identify  selection  criteria  and  nominate 
technical  experts  who  could  serve  as  assessors  and 
evaluators . 

2.1.7.  Recommend  any  other  special  technical 
requirements  which  should  be  considered  as  necessary 
for  accreditation  in  this  testing  field. 

2.2.  Application  for  Accreditation. 

2.2.1.  Any  laboratory  may  apply  for  accreditation  in 
any  conformance  testing  program  following  the 
instructions  provided  in  notices  announcing  the  formal 
establishment  of  the  program.  Appendix  B provides  a 
sample  application  for  a Testing  Laboratory. 

2.2.2.  Upon  receipt  of  the  laboratory's  application, 
the  Certification  Body  should: 

2. 2. 2.1.  acknowledge  receipt  of  the  application; 

2.  2. 2. 2.  request  further  information,  if 
necessary; 

2. 2. 2. 3.  confirm  any  payment  of  any  fees  before 
proceeding  with  the  accreditation  process;  and 

2. 2. 2. 4.  specify  the  next  steps (s)  in  the 
accreditation  process. 

2.2.3.  In  accepting  an  application  from  a foreign- 
based  laboratory,  consideration  should  be  given  to  the 
policy  of  the  host  government  regarding  the  acceptance 
of  test  data  from  laboratories  accredited  by  other 
domestic  or  foreign  accreditation  systems. 

3.  TESTING  METHODS 

3.1.  Commonly  accepted  conformance  test  methods. 
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3.1.1.  Falsification  Testing.  Types  of  standards  which 
currently  employ  this  method  include: 

3 .1.1.1.  programming  language  standards; 

3.1. 1.2.  graphics  standards;  and 

3 • 1 • 1 . 3 . text  standards . 

3.1.2.  Proofs  of  Correctness  Testing.  This  is  a more 
difficult  process,  but  the  more  desired  approach 
because  of  its  completeness.  To  date,  no  known 
conformance  testing  program  is  able  to  offer  proof  of 
correctness. 

3.2.  Guidelines  for  development  of  test  method  approaches. 
The  test  method  should  be  specified  in  a way  that  is  easy  to 
understand  and  to  implement.  The  resulting  test  suite 
should  be  useful  to  implementors.  The  following  steps  are 
recommended  when  developing  a test  method: 

3.2.1.  Management  Plan.  Describes  the  overall  scheme 
for  the  testing.  It  should  include  a specification  of 
the  technical  objectives  to  be  achieved,  a description 
of  the  test  strategy,  and  an  outline  of  the  procedures 
to  be  used  for  testing.  This  management  plan  should  be 
publicly  available. 

3.2.2.  Non-Testable  Features.  Although  the  objective 
is  always  to  test  all  functions  in  a standard,  there 
are  sometimes  parts  of  a standard  which  are  non- 
testable.  For  example,  some  error  conditions, 
depending  on  their  nature,  cannot  be  included  in  a 
standardized  test  method  to  check  conformance.  A list 
of  all  areas  explicitly  not  tested  should  be  available 
as  part  of  the  documentation  of  the  test  method. 

3.2.3.  Test  Strategy.  A full  description  of  the  test 
strategy  to  be  used  for  the  standard  should  be  produced 
and  made  publicly  available.  This  should  include  a 
specification  of  the  structure  of  the  proposed  test 
suite,  an  indication  of  the  number  of  tests,  and  an 
identification  of  the  elements  of  the  standard  to  be 
tested. 

3.2.4.  Test  Procedures.  The  test  method  should  be 
automated  and  include  consideration  of  the  procedures 
used  to  carry  out  testing.  The  setting  up  and 
execution  of  the  test  suite  on  a system  under  test,  and 
the  analysis  of  the  output  produced  should  be  carried 
out  in  accordance  with  clearly  defined  procedures. 
These  procedures  should  be  documented  in  a manual  to 
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accompany  the  test  suite,  and  should  give  criteria  for 
the  analysis  of  all  acceptable  outputs  for  each  test  in 
the  test  suite.  Specific  assumptions  for  failure  of 
test  should  also  be  stated. 

3.2.5»  Levels  of  Testing.  The  development  of  the 
test  method  should  take  into  account  all  possible 
levels  and/or  options  specified  in  the  standard.  The 
test  method  should  allow  for  modularity  so  that  tests 
for  a specific  level  or  option  can  be  easily  selected. 

3 0 3 . Existing  test  methods. 

The  following  areas  should  be  examined  when  selecting  the 
appropriate  test  methods 

3.3.1.  Standard  specification: 

3. 3. 1.1.  Definition  of  conditions  for  conformance 
with  standards  (conformance  clauses) . 

3 .3.  1.2.  Specification  requirements.  The 
requirements  of  the  standard  should  be  evaluated 
to  identify  the  test  method  necessary  for 
determining  conformance.  This  method  might 
include  tests  for  determining  that  the 
implementation : 

3. 3. 1.2.1.  rejects  illegal  structures; 

3 . 3 . 1 . 2 . 2 . accepts  data  and  other  constructs 
permitted  by  the  standards,  and  processes 
them  in  accordance  with  the  standard; 

3. 3. 1.2. 3.  does  not  include  extensions  to  a 
standard; 

3. 3. 1.2. 4.  properly  flags  usage  of  standard 
requirement  options  or  levels; 

3. 3.  1.2. 5.  processes  implementation 
dependent  specifications  in  accordance  with 
the  standard; 

3. 3. 1.2. 6.  processes  optional  implementation 
features  of  the  standard  correctly. 

3 o 3 c 2 . Written  test  procedures  supporting  the  test 
suite  to  identify: 

3. 3. 2.1.  test  objective; 

3 . 3 e 2 . 2 . personnel  responsible  for  test ; 
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3 . 3 . 2 . 3 . 

necessary  equipment? 

3. 3. 2. 4. 

materials,  e.g.,  support  software? 

3. 3. 2.5. 

input  data? 

3 . 3 . 2 . 6 . 

expected  output? 

3. 3.2. 7. 

constraints  ? 

3 . 3 . 2 . 8 . 

required  test  preparation,  setup? 

3 . 3 . 2 . 9 . 

operator  actions,  console  setups?  and 

3.3,2.10. 

step-by-step  execution  instructions . 

3.3.3.  The  test  suite  to  see  if  it  is: 

3 . 3 . 3 . 1 . easy  to  understand  and  manage  ? 

3. 3. 3. 2.  portable  between  different  hardware 
configurations  and  designed  to  take  into  account 
any  implementation-dependent  criteria  allowed  by 
the  standard? 

3. 3. 3. 3.  designed  in  such  a way  that  there  are  no 
restrictions  in  using  the  test  method  or 
implementations  on  the  size  machine  on  which  it 
can  be  executed. 

3.3.4.  Documentation  Requirements  for  inclusion  of: 


3. 3. 4.1. 

description  of  test  cases? 

3 . 3 . 4 . 2 . 

notation  used  to  define  test? 

3. 3. 4. 3. 
and 

test  method  implementation/user  guides  ? 

3.3. 4. 4. 

evaluation  guides  for  qualifying  pass, 

fail,  or  inconclusive  test  verdicts. 

3.3.5.  Requirements,  if  any,  for  specially  designed 
test  equipment  training. 

3.4.  Test  Suite  Structure. 

The  following  guidelines  should  be  followed  when  writing 
test  suites : 

3.4.1.  Clear  documentation  should  be  incorporated  into 
the  tests  and  this  documentation  should  include: 
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3. 4. 1.1.  comments,  which  are  clear  and  concise. 
No  redundant  comments  should  be  included. 

3.4. 1.2.  a reference  to  the  clauses  in  the 
standard  which  are  under  test. 


3 .4.1. 3.  clear  statements  of  any  assumptions  made 
in  the  test  suite  design. 

3.4. 1.4.  a list  of  any  tests  which  are  performed 
for  information  only,  and  will  not  be  used  to 
determine  conformance. 


3.4.2.  The  test  suite  should  be  split  into  a series  of 
independent  test  programs  or  scenarios,  such  that  each 
test  program  tests  only  one  ‘specific  requirement  of  the 
specification  document. 


3.4.3.  Each  self-contained  test  should  use  the  same 
format,  and: 

3. 4. 3.1.  list  all  test  tools  necessary? 

3. 4. 3. 2.  provide  the  testing  environment  data, 
including  display  of  setup  instructions  to 
operator  ? 

3 . 4 . 3 . 3 . include  program  documentation  section  ? 

3. 4. 3. 4.  initialize  its  own  workspace? 


3. 4. 3. 5.  perform  the  test? 

3 .4. 3. 6.  provide  a verification  section? 

3. 4. 3. 7.  check  whether  passed/failed? 


3. 4. 3. 8.  report  result? 

3. 4. 3. 9.  have  the  pass/fail  path  clearly  defined? 

3.4.3.10.  include  a simple  comment  describing  the 
task  performed  by  the  test  and  a reference  to  the 
section  of  the  standard  under  test?  and 


3.4.3.11.  be  kept  as  simple  as  possible. 

3.4.4.  Only  simple  features  of  the  test  support  code 
should  be  used. 


14 


3.4.5.  Translating  the  Test  Suite  to  Other  Languages. 
If  possible  and  when  applicable,  the  test  suite  should 
be  written  in  a common  format,  which  can  then  be 
converted,  by  the  use  of  translators,  to  each  of  the 
programming  languages.  Updates  to  the  test  suites 
across  languages  must  be  kept  in  line  at  all  times  by 
strict  use  of  the  change  control  procedures. 


3.5.  Maintenance  of  Test  Methods 

3.5.1.  Responsible  Party.  The  Certification  Body  is 
responsible  to  ensure  maintenance  of  current  test 
methods,  authorize  changes,  define  the  procedures  for 
changes,  and  indicate  when  any  changes  become  effective 
for  validation  purposes. 

3.5.2.  Procedures.  A published  change  control 
procedure  should  be  established  for  both  the  test 
software  and  the  associated  documentation.  The  change 
control  procedures  should  provide  a formal  mechanism 
for  the  following: 


3 . 5 . 2 . 1 . 

3 . 5 . 2 . 2 . 

3 c 5 . 2 . 3 . 
and 


reporting  errors  in  the  test  software; 
logging  changes  to  the  test  software; 
reporting  errors  in  the  documentation; 


3. 5. 2. 4.  logging  changes  to  the  documentation. 


4.  CERTIFICATION  AND  REPORTING  CONSIDERATIONS 


4.1.  Certification  Body. 

4.1.1.  General  Requirements. 

4. 1.1.1.  Access  to  the  Certification  Body  shall 
not  be  conditional  upon  membership  in  any 
association  or  group,  nor  shall  there  be  undue 
financial  conditions  to  restrict  participation. 

4. 1.1. 2.  The  procedures  under  which  the  Body 
operates  should  be  administered  in  a 
nondiscriminatory  manner. 

4. 1.1. 3.  ICST  will  identify  the  members  on  the 
Certification  Body. 

4.1.2.  Responsibilities. 
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4. 1.2.1.  Approve  test  methods  to  be  used  by  the 
testing  laboratories  when  they  are  not  included  in 
the  standard  itself.  Ensure  the  test  method  is 
technically  correct  to  the  FIPS  before  using  it  as 
the  basis  for  issuing  certificates. 

4 . 1 o 2 o 2 » Collaborate  with  the  testing  laboratory 
in  drafting  a contract  for  testing  services  to 
include  regulations  which  define  rights  and 
obligations  in  detail. 

4 . 1 . 2 . 3 . Determine  accreditation  criteria  for 
testing  laboratories  and  keep  lists  of  all 
recognized  to  do  conformance  testing.  -Since 
technical  requirements  for  laboratory 
accreditation  are  specific  for  each  standard, 
technical  requirements  for  accreditation  should  be 
developed  using  expert  advice  in  that  given  field. 
Appendix  C provides  a Model  Certificate  for 
Accreditation . 

4. 1.2. 4.  Notify  any  testing  laboratory  that  it  is 
no  longer  accredited. 

4. 1.2.5.  Issue  certificates  at  the  request  of  the 
Client,  based  on  the  certification  criteria 
defined  by  the  Certification  Body. 

4.1.2.6.  When  applicable,  define  the  requirements 
and  procedures  of  a Manufacturer's  Declaration  of 
Conformance  Testing  Program. 

4. 1.2. 7.  Define  Client/testing  laboratory  dispute 
procedures.  Arbitrate  appeals  in  cases  of 
dispute.  Include  procedures  for  disputed  tests  as 
well  as  disputed  testing  results. 

4 .1.2. 8.  Determine  if  there  is  a need  for  a 
Certified  Computer  Products  List.  If  there  is  a 
need  (possibly  on  a case  by  case  basis) , identify 
the  maintainer  and  what  information  should  be 
reported. 

4 .1.2.9.  Determine  what  process  should  be 
followed  in  recognizing  other  published  Certified 
Computer  Products  Lists. 

4.1.2.10.  Notify  the  public  by  advertising  new 
conformance  testing  programs  when  the  technical 
requirements  and  test  methods  are  developed  for 
the  standard (s) . Publish  a notice  in  the  FEDERAL 
REGISTER  announcing  the  establishment  of  the 
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program,  requirements,  or  methods, 
will : 


The  notice 


4.1.2.10.1.  identify  the  scope  of  the 
program  ,*  and 


4.1.2.10.2.  advise  how  to  request  validation 
services. 


4.1.2.11.  Develop  criteria  and  procedures  for 
evaluating,  modifying  and  accepting  test  methods 
for  use  in  conformance  testing  to  the  standards. 
Keep  in  mind  such  test  methods  sources  as: 


4.1.2.11.1. 

4.1.2.11.2. 

4.1.2.11.3. 

4.1.2.11.4. 


industry ; 
government ; 
other  countries  ? and 
universities. 


4.1.2.12.  Publish  the  effective  date  for 
modifications  to  test  methods. 


4.2.  ICST  Role  in  Conformance  Testing  As  Government  Agent. 

4.2.1.  Assist  and  educate  the  agencies  with  regard  to 
the  scope  and  purpose  of  the  test  method  as  a measure 
of  product  quality. 

4.2.2.  Assist  agencies  in  developing  solicitation 
document  requirements  with  regard  to  product  testing 
for  determining  conformance  to  standards. 

4.2.3.  Identify  publicity  and  announcement  sources  for 
informing  national  and  international  industry, 
Government  and  product  users  on  the  availability  of 
conformance  test  methods  and  validation  services. 


4.2.4.  Work  with  other  government  agencies,  as 
required,  to  incorporate  requirements  within  the 
procurement  regulations  (i.e.,  General  Services 
Administration's  Federal  Information  Resources 
Management  Regulations)  concerning  use  of  conformance 
testing  and  certification  of  products  offered  to  the 
Government . 

4.3.  Role  of  Standards  Bodies  (National  and  International). 

4.3.1.  Consider  conformance  testing  aspects  in  the 
development  of  standards. 
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4.3.2.  Provide  advice  and  give  technical  committee  or 
subcommittee  recommendations  for  the  approval  of  a test 
method. 

4.3.3.  Provide  technical  advice  when  necessary,  on 
interpreting  a standard  under  conflict. 

4.4.  Test  Report. 

4.4.1.  Content. 

4. 4. 1.1.  Includes  a complete  test  report  and  a 
statement  citing  laboratory  accreditation. 

4 .4. 1.2.  Includes  complete  identification  of 
system  under  test  (SUT) . 

4. 4. 1.3.  Provides  a description/identification  of 
the  test  environment,  and  any  incidents  contrary 
to  routine  process. 

4.4.2.  Format.  The  following  guidelines  should  be 
followed  when  writing  the  software  tools  to  produce  the 
test  program  reports: 

4.4.2. 1.  The  volume  of  output  produced  by  each 
test  program  should  be  kept  as  concise  as 
possible.  Tabular  summaries  are  encouraged. 

4. 4. 2. 2.  The  following  details  should  be  given  in 
the  test  report  for  each  test:  test  number,  brief 
description  of  the  test,  reference  to  the  standard 
and  implementation  under  test  (IUT) , and  message 
indicating  pass/ fail/ inconclusive , or  not 
executed. 

4. 4. 2. 3.  Test  discrepancy  summary  complete  enough 
to  repeat  the  process. 

4.4.3.  Style  for  Test  Reporting. 

4. 4. 3.1.  Each  test  report  should  provide  a 

summary  of  the  results  giving  the  following 
information:  number  of  tests  passed,  number  of 

tests  failed,  number  of  inconclusive  tests,  number 
of  "information  only"  tests,  number  of  tests  not 
executed,  and  total  number  of  tests. 

4. 4. 3. 2.  Wherever  possible,  the  output  of  test 
programs  should  be  structured  in  a way  which  is 
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easy  to  check  both  manually  and  automatically, 
e.g.,  tabular. 

4 .4. 3. 3.  If  test  programs  produce  visual  output 
on  a graphics  display,  then  this  output  should  be 
kept  as  simple  as  possible  to  ease  the  checking 
process,  while  still  examining  all  necessary 
features. 

4. 4. 3. 4.  All  tests  which  produce  visual  output 
should  be  accompanied  by  a comprehensive  check 
list  in  the  form  of  an  operator  script  and  a set 
of  reference  samples. 

4 .4.3. 5.  Particular  care  and  attention  should  be 
paid  to  the  arrangement  of  the  test  report, 
especially  with  regard  to  presentation  of  the  test 
data  and  ease  of  assimilation  by  the  reader.  The 
format  should  be  carefully  and  specifically 
designed  for  each  type  of  test  carried  out,  but 
the  headings  should  be  standardized  as  far  as 
possible. 

4.5.  Certificate  of  Validation  Format,  Content  and 

Presentation . 

4.5.1.  Identifies  the  standard (s)  tested. 

4.5.2.  Includes  a statement  of  the  fact  that  the 

product (s)  or  service (s)  meets  the  standard (s)  or  an 
applicable  portion (s)  thereof,  clearly  identifying  that 
which  is  being  certified. 

4.5.3.  Includes  a clear  reference  to  the  testing 

laboratory  and  test  report  produced. 

4.5.4.  States  the  product  name  and  version  tested. 

4.5.5.  The  certificate  only  applies  to  the  product 

version  operating  in  the  SUT. 

4.5.6.  Certificate  validity  depends  on  the  set  of 

standards,  and  the  specified  period  of  time. 

4.6.  Applicability  of  Certificate  of  Validation  on  product 
tested.  Some  of  the  criteria  to  consider  include: 

4.6.1.  The  system  configurations  that  require  testing 
for  the  product  for  which  conformance  is  claimed. 

4.6.2.  The  number  and  type  of  errors  discovered  during 
testing. 
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4.6.3.  Use  of  the  validated  product  under  different 
operating  environments  e.g.,  different  hardware 
configurations  and  operating  systems. 

4.6.4.  New  software  releases. 

5.  TESTING  ADMINISTRATION 

5.1.  General  Guidelines  for  Test  Method  Administration. 

5.1.1.  Test  methods  need  to  be  approved  by  the 
Certification  Body  under  technical  guidance  of  the 
responsible  standards  committee. 

5.1.2.  Test  methods  for  which  conformance  testing  is 


offered 

adequate 

are  made  available 
specified  conditions. 

to 

everybody 

through 

5.1.3. 

binding 

Test  methods  offered 
on  conformance  testing. 

are 

official 

for 

and 

5.1.4. 

The  same  test  methods 

derived 

from 

the 

standards  specification  should 
approved  testing  laboratories. 

be 

used  by 

all 

ICST 

5.1.5. 

Test  method  upon  which 

certification 

is 

based 

is  s 


5. 1.5.1.  acceptable  to  buyer  or  governmental 
regulator  if  mandated. 

5. 1.5. 2.  used  in  its  entirety  unless  limitations 
are  fully  disclosed. 

5. 1.5. 3.  is  based  on  a FIPS  (which  is  based  on  a 
national  or  international  standard)  available  to 
the  public. 

5. 1.5.4.  suitable  for  use  as  a basis  for 
conformance  testing. 

5.1.6.  Declaration  of  conformance  is  indicated  by  a 
statement,  certificate,  label  or  mark  which: 

5. 1.6.1.  includes  information  on  the  product, 
standards,  producer,  and  procedures  used. 

5. 1.6.2.  presents  clearly  the  intent  of  the 
declaration. 
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5.2.  Criteria  and  Procedures  for  Alternative  Testing 
Sources . 


5.2.1.  Manufacturer's  declaration  of  conformance  (Self 
testing) . Requirements  for  Vendor  include: 

5. 2. 1.1.  manufacturer's  quality  management  system 
and  documentation; 

5. 2. 1.2.  technically  appropriate  resources  for 
testing  and  inspection?  - 

5. 2. 1.3.  use  of  a test  method  approved  by  the 
Certification  Body; 

5. 2. 1.4.  retention  of  complaint  records  against 
product  or  service?  and 

5. 2. 1.5.  any  dispute  resolution  is  in  accordance 
with  the  Certification  Body  for  that  standard. 

5.2.2.  Third-party  testing.  Requirements  include: 

5. 2. 2.1.  products  to  be  tested  are  fully 
identified. 

5. 2. 2. 2.  procedures  by  which  the  laboratory  is 
operated  include  provision  for: 

5. 2. 2. 2.1.  fiscally  responsible  management 
and  appropriately  trained  staff. 

5. 2. 2. 2. 2.  participation  on  a non- 
discriminatory  basis. 

5. 2. 2. 2. 3.  resource  commitment  to  both 
initial  and  continuing  validation  activities. 

5. 2. 2. 2. 4.  selection  and  retention  of 
qualified  testing  and  inspection  services. 

5. 2. 2. 2. 5.  notification  of  participants  on 
changes  in  standards  and  procedures. 

5. 2. 2. 2. 6.  confidentiality  of  proprietary 
information. 

5. 2. 2. 2. 7.  maintenance  of  records. 

5. 2. 2. 2. 8.  safeguarding  use  of  official 
declaration  of  conformance. 
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5 . 2 e 2 . 3 . Documentation  required  by  the  laboratory 
includes: 


5 .2. 2 .3.1.  availability  of  published  program 
directory  listing  products,  standards, 
licensees,  and  other  parties. 

5.2. 2. 3. 2.  file  of  legally  binding 
agreements  for  licensees. 

5.2.2. 3.3-.  availability  of  published 
statement  of  operating  procedures. 

5.2.2 .4.  Approved  by  Certification  Body. 

5.2.3.  Government-managed  testing.  The  same  criteria 
and  procedures  for  third-party  testing  would  apply. 

5.3.  Associated  Costs. 

Identify  costs  associated  with  developing  and  maintaining 
conformance  test  methods  and  responsible  source  of  funding. 
Typically  costs  associated  with  conformance  testing  will  be 
on  a cost-reimbursable  basis. 

5.4.  Resource  and  overhead  requirements.  Examine: 

5.4 .1.  Where  and  what  type  of  facilities  are  required 
to  perform  validations.  Considerations  of  competence, 
impartiality,  and  integrity  are  fundamental  to  the 
acceptability  of  the  testing  laboratory.  Clients  of 
the  testing  services  must  be  guided  by  this  fact  in 
making  the  choice  of  testing  laboratories  that  they 
employ. 


5 . 4 o 2 . What  staff,  equipment  and  office  facilities  are 
required  to  support  a validation  service. 

5.4.3.  What  are  the  retention  and  storage  requirements 
for  validation  materials,  project  folders  and  financial 
accounting  records. 

5.4.4.  What  is  the  anticipated  volume  of  Clients  and 
tests  conducted  over  a year. 

5.5.  Criteria  and  Procedures  for  Requesting  Validation 
Services.  This  section  presents  typical  procedures  which 
could  be  used  for  requesting  and  scheduling  validations,  and 
the  table  below  provides  only  example  timeframes  associated 
with  first  time  validation.  These  times  will  vary  according 
to  complexity  of  tests  requested,  and  number  of  tests  being 
performed. 
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SCHEDULE  EXPECTATIONS  FOR  VALIDATION 


T - 120 

T - 90 
T - 60 

T - 30 
TEST 
T + 4~7 
T + 60 


Contact  approved  testing  laboratory, 
Negotiate  agreement;  vendor  testing 

Submit  disputed  tests,  if  any 

Prevalidation  test  information/material 
received  by  the  testing  laboratory 

Resolve  all  outstanding  issues 

Laboratory  testing 

Testing  complete 

Final  Conformance  Summary  Report  available 


Key 

T = Testing  Day  1;  ' or  ' + ' = Before/Beyond  Test  Start 


Table  I. 


5.5.1.  Requesting  validation.  The  Client  provides  the 
following  information/material  to  the  approved  testing 
laboratory  in  a Letter  of  Request: 

5. 5. 1.1.  desired  and  alternate  month  of  testing; 

5. 5. 1.2.  product  identification,  including 
version; 

5. 5. 1.3.  host  and  target  configurations, 
operating  systems,  program  language  and  other 
appropriate  information  necessary  for  the  testing 
laboratory  to  carry  out  the  test; 

5. 5. 1.4.  site  locations  and  addresses; 

5. 5. 1.5.  a point  of  contact  for  validation, 
including  full  name,  address,  and  telephone 
number ; and 

5.5. 1.6.  which  test(s)  the  Client  desires  to 
take . 

5.5-.  2.  Revalidation  requests.  The  Client  requests 
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revalidation  by  the  appropriate  testing  laboratory. 
The  request  includes: 

5. 5. 2.1.  date  of  current  certificate  of 
conformance . 

5c5o2«2.  a statement  clearly  confirming  the 
computer  product  which  is  certified  was  not 
modified  from  the  computer  product  described  in 
the  Conformance  Summary  Report. 

5.5.3.  Revalidating.  When  a modification  which  is 

beyond  continuing  maintenance  has  been  made  to  the 
computer  product  previously  certified,  it  should  be 
revalidated. 

5.5.4c  Scheduling  Validations.  These  times  apply  to 
the  Client's  responsibilities  directly  with  the  testing 
laboratory;  the  specific  month  in  which  the  testing 
takes  place  is  established  by  the  laboratory. 

5. 5. 4.1.  Any  prevalidation  information/material 
should  be  received  60  days  prior  to  the  month  of 
the  scheduled  validation. 

5. 5. 4.2.  For  products  not  previously  tested, 
request  for  validation  should  be  submitted  at 
least  120  days  prior  to  the  desired  testing  month. 
Data  collection  dates  should  not  be  established 
for  first  time  validations  until  after  a 
successful  review  of  prevalidation  materials  has 
been  made  by  the  testing  laboratory. 

5.5.4 .3.  Requests  for  changes  to  an  existing  test 
date  should  be  submitted  no  later  than  60  days 
prior  to  that  date. 

5. 5. 4. 4.  Priorities.  Priority  is  given  to  the 
scheduled  annual  validations.  Requests  from 
Clients  for  special,  procurement-related 
validations  are  given  second  priority.  Third 
priority  is  given  to  requests  for  official 
validation  of  new  products.  All  other  requests 
are  given  a lesser  priority. 

5.5.5.  Appeal  Process. 

5. 5. 5.1.  Disputed  tests.  If  a Client  believes 
any  tests  are  in  error,  or  not  appropriate,  the 
Client  contacts  the  testing  laboratory,  and 
provides  associated  justification  for  the  dispute. 


24 


A copy  of  this  letter  should  be  forwarded  to  the 
Certification  Body  as  well. 

5. 5. 5. 2.  The  Client  follows  the  procedures 
defined  by  that  Certification  Body  which  approved 
the  testing  laboratory  for  any  appeals  over 
disputed  tests  or  disputed  validation  testing 
decisions  by  the  Accredited  Testing  Laboratory. 

6 . TESTING  LABORATORIES 

6.1.  Conditions  for  Accreditation.  The  conditions  for  a 
laboratory  to  become  accredited  and  maintain  accreditation 
depends  on  the  testing  requirements.  Laboratory 
accreditation  may  require  the  Testing  Laboratory  tot 

6.1.1.  be  assessed  and  evaluated  initially  and  on  a 
periodic  basis. 

6.1.2.  demonstrate,  on  request,  that  it  is  able  to 
perform  the  tests  representative  of  those  for  which  it 
is  seeking  accreditation. 

6.1.3.  pay  any  relevant  fees. 

6.1.4.  participate  in  proficiency  testing  as  required. 

6.1.5.  be  capable  of  performing  the  tests  for  which  it 
is  accredited  according  to  the  latest  version  of  the 
test  method  within  one  year  after  its  publication  or 
within  the  time  limit  specified  by  the  Certification 
Body. 


6.1.6.  limit  the  representation  of  the  scope  of  its 
accreditation  to  only  those  tests  or  services  for  which 
accreditation  is  granted. 

6.1.7.  limit  all  its  test  work  or  services  for  Clients 
to  available  capacity. 

6.1.8.  limit  advertising  of  its  accredited  status  to 
letterheads,  brochures,  test  reports,  and  professional, 
technical,  trade  or  other  laboratory  services 
publications . 

6.1.9.  inform  its  Clients  that  the  laboratory's 
accreditation  or  any  of  its  test  reports  in  no  way 
constitutes  or  implies  product  certification,  approval, 
or  endorsement  by  ICST. 

6.1.10.  maintain  records  of  all  actions  taken  in 
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response  to  testing  complaints  for  a minimum  of  one 
year. 


6cl.ll.  maintain  an  independent  decisional 
relationship  between  itself  and  its  Clients, 
affiliates,  or  other  organizations  so  that  the 
laboratory's  capacity  to  render  test  reports 
objectively  and  without  bias  is  not  adversely  affected. 

6.1.12.  report  within  30  days  any  major  changes 
involving  the  location,  ownership,  management 
structure,  authorized  representation,  approved 
signatories,  or  facilities  of  the  laboratory. 

6.2.  Criteria  for  Accrediting  Testing  Laboratories. 

The  criteria  address  a laboratory's  quality  system,  staff, 
facilities  and  equipment,  test  methods  and  procedures, 
records,  and  test  reports.  At  a minimum  following  ISO/XEC 
Guide  25,  the  criteria  below  should  be  considered  in  order 
for  a testing  laboratory  to  be  accredited: 

6.2.1.  Quality  System 

6.2.1.L  The  laboratory  should  operate  an 
internal  quality  assurance  program  appropriate  to 
the  type,  range,  and  volume  of  work  performed. 
The  quality  assurance  program  should  be  documented 
in  a quality  assurance  manual  which  is  available 
for  use  by  the  laboratory  staff.  The  manual 
should  be  maintained  relevant  and  current  by  a 
responsible  member  of  the  laboratory  staff.  A 
person  or  persons  having  responsibility  for 
quality  assurance  within  the  laboratory  should  be 
designated  by  the  laboratory  management  and  have 
direct  access  to  top  management. 

6.2. 1.2.  The  quality  assurance  manual  should 
contain  information  regarding: 

6. 2. 1.2.1.  the  structure  of  the  laboratory 
(organizational  charts) ? 

6.2. 1.2.2.  the  operational  and  functional 
duties  and  services  pertaining  to  quality,  so 
that  each  person  concerned  will  know  the 
extent  and  the  limits  of  his  responsibility; 

6.2. 1.2. 3.  general  quality  assurance 
procedures ; 

6. 2. 1.2. 4.  quality  assurance  procedures 
specific  for  each  test,  as  appropriate; 
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6. 2. 1.2. 5.  where  appropriate,  proficiency 
testing,  use  of  reference  material,  etc; 

6 .2.1. 2. 6.  satisfactory  arrangements  for 
feedback  and  corrective  action  whenever 
testing  discrepancies  are  detected? 

6. 2. 1.2. 7.  procedure  for  dealing  with 
technical  complaints. 

6. 2. 1.3.  The  quality  system  should  be 
systematically  and  periodically  reviewed  by,  or  on 
behalf  of  management  to  ensure  the  continued 
effectiveness  of  the  arrangements,  and  corrective 
action  initiated.  Such  reviews  should  be  recorded 
together  with  details  of  any  corrective  action 
taken . 

NOTE:  In  small  laboratories,  the  quality  system 

may  fulfill  the  requirements  of  this  clause  in  a 
simplified  way. 

6.2.2.  Staff 

6. 2. 2.1.  Staff  should  have  the  necessary 
education,  training,  technical  knowledge  and 
experience  for  their  assigned  functions. 

6. 2. 2. 2.  There  should  be  a job  description  for 
each  senior  technical  position  category,  which 
includes  the  necessary  education,  training, 
technical  knowledge  and  experience. 

6 . 2 . 2 . 3 . The  proportion  of  supervisory  to  non- 
supervisory  staff  should  be  such  as  to  ensure 
adequate  supervision. 

6. 2. 2. 4.  Suitable  staff  should  be  nominated  to 
deputize  for  the  senior  technical  and  quality 
system  management  staff  in  their  absence. 

6.  2.  2.  5.  Information  on  the  relevant 
qualifications,  training,  and  experience  of  the 
technical  staff  should  be  maintained  by  the 
laboratory. 

NOTE:  In  small  laboratories,  one  person  may 

fulfill  more  than  one  function. 

6.2.3.  Testing  Equipment 
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6. 2. 3.1.  The  testing  laboratory  should  have 
access  to  all  items  of  equipment  required  for 
correct  performance  of  the  tests  for  which  it  is 
recognized. 

6 . 2 . 3 . 2 . All  equipment  should  be  properly 
maintained  to  ensure  protection  from 
deterioration.  Instructions  for  a proper 
maintenance  procedure  for  those  items  of  equipment 
which  require  periodical  maintenance  should  be 
available. 


6 . 2 o 3 . 3 . Any  item  of  the  equipment  which  has  been 
subjected  to  overloading  or  mishandling,  or  which 
gives  suspect  results,  or  has  been  shown  to  be 
defective,  should  be  taken  out  of  service  and 
clearly  labelled  until  it  has  been  repaired  and 
then  shown  by  test  to  be  performing  its  function 
satisfactorily. 

6. 2. 3. 4.  Records  should  be  maintained  on  each 
major  item  of  equipment.  Each  record  should 
include: 

6. 2.  3.  4.1.  the  name  of  the  item  of 

equipment. 


6 . 2 . 3 . 4 . 2 . the  manufacturer ' s name  and  type 
identification  and  serial  number. 

6. 2. 3. 4. 3.  date  received  and  date  placed  in 
service. 

6. 2.3.4. 4.  current  location,  where 
appropriate. 

6. 2. 3. 4. 5.  details  of  maintenance. 

6.2.4.  Test  methods  and  procedures 

6.2.4. 1.  The  testing  laboratory  should  have 
adequate  documented  instructions  on  the  use  and 
operation  of  all  relevant  equipment,  on  the 
handling  and  preparation  of  test  items  (where 
applicable) , and  on  standard  testing  techniques, 
where  the  absence  of  such  instructions  could 
jeopardize  the  efficacity  of  the  testing  process. 
All  instructions,  standards,  manuals,  and 
reference  data  relevant  to  the  work  of  the  testing 
laboratory  should  be  maintained  up-to-date  and  be 
readily  available  to  the  staff. 
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6. 2. 4. 2.  The  testing  laboratory  should  use 
methods  and  procedures  required  by  the 
specification  against  which  the  test  items  are  to 
be  tested.  The  specification  should  be  available 
to  staff  performing  the  test. 

6. 2. 4. 2.1.  all  manual  calculation  and  data 
transfers  should  be  subject  to  appropriate 
checks . 

6 . 2 . 4 . 2 . 2 . where  these  results  are  derived 
by  electronic  data  processing  techniques,  the 
stability  of  the  system  should  be  such  that 
the  accuracy  of  the  results  is  not  affected. 
This  implies  an  ability  to  detect 
malfunctions  in  the  hardware  during  program 
execution  and  to  take  appropriate  action. 

6.2.5.  Environment 

6. 2. 5.1.  The  environment  in  which  the  tests  are 
undertaken  should  not  invalidate  the  test  results 
or  adversely  affect  the  required  accuracy.  The 
testing  premises  should  be  protected  as  required 
from  conditions  such  as  excessive  temperature, 
dust,  moisture,  steam,  vibration,  electromagnetic 
disturbance  and  interference,  and  should  be 
maintained  accordingly.  There  should  be 
sufficient  space  to  limit  the  risk  of  damage  or 
danger  and  to  allow  operators  to  make  practical 
and  precise  movements.  The  premises  should  have 
the  equipment  and  energy  sources  needed  for  the 
testing. 


6. 2. 5. 2.  Access  to  and  use  of  all  test  areas 
should  be  controlled  in  a manner  appropriate  to 
their  designated  purpose,  and  entry  by  persons 
external  to  the  laboratory  should  be  defined. 

6. 2 c 5. 3.  Adequate  measures  should  be  taken  to 
ensure  good  housekeeping  in  the  testing 
laboratory. 

6.2.6.  Records 


6. 2. 6.1.  The  testing  laboratory  should  maintain  a 
record  system  to  suit  its  particular  circumstances 
and  comply  with  any  existing  regulations.  It 
should  retain  on  record  all  original  observations, 
calculations  and  derived  data,  and  the  final  test 
report  for  an  appropriate  period  as  designated. 
The  records  for  each  test  must  contain  sufficient 
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information  to  permit  satisfactory  repetition  of 
the  test. 

NOTE:  In  some  countries  it  may  be  necessary  to 

maintain  records  for  a period  specified  by  law. 

6c 2 060 2.  All  records  and  test  reports  should  be 
held  secure  and  in  confidence  to  the  Client, 
unless  otherwise  required  by  law. 

6.2.7.  Test  reports 

60  2. 7.1.  The  work  carried  out  by  the  testing 
laboratory  should  be  covered  by  a report  which 
accurately,  clearly,  and  unambiguously  presents 
the  test  results  and  all  other  relevant 
information. 

6. 2. 7. 2.  Each  test  record  should  include  at  least 
the  following  information: 

60 2. 7. 2 .1.  name  and  address  of  testing 
laboratory? 

6. 2. 7. 2. 2.  unique  identification  of  report 
(such  as  serial  number) , and  of  each  page  of 
the  report  (sequential  numbering) ? 

6.2. 7.2.3.  name  and  address  of  Client; 

6. 2. 7. 2. 4.  description  and  identification  of 
the  test  item? 

6. 2. 7. 2 .5.  date(s)  of  performance  of  test, 
as  appropriate? 

6. 2. 7. 2. 6.  a statement  to  the  effect  that 
the  test  results  relate  only  to  the  items 
tested? 

6. 2. 7. 2. 7.  identification  of  the  test 
specification,  method,  and  procedure? 

6. 2. 7. 2. 8.  description  of  sampling 
procedure,  where  relevant? 

6. 2. 7 .2. 9.  any  deviations,  additions  to,  or 
exclusions  from  the  test  specification,  and 
any  other  information  relevant  to  a specific 
test ; 
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6.2.7.2.10.  disclosure  of  any  non-standard 
test  method  or  procedure  utilized? 

6.2.7.2.11.  derived  results,  supported  by 
tables,  graphs,  sketches,  and  photographs  as 
appropriate,  and  any  failures  identified? 

6.2.7.2.12.  a statement  on  test  performance 
uncertainty  where  relevant? 

6.2.7.2.13.  a signature  and  title  of 
person (s)  accepting  technical  responsibility 
for  the  test  report  and  date  of  issue? 

6 c 2 . 7 . 2 . 14 . a statement  that  the  report 
should  not  be  reproduced  except  in  full  o 
without  the  approval  of  the  testing 
laboratory. 

6 . 2 . 7 . 3 . Corrections  or  additions  to  a test 
report  after  issue  should  be  made  only  by  a 
further  document  suitably  marked,  e.g., 
"Supplement  to  test  report  serial  number  . . . (or 
as  otherwise  identified),"  and  should  meet  the 
relevant  requirements  of  the  preceding  paragraphs. 

6.3.  Testing  Laboratory  Assessment.  (Figure  1,  below,  is  a 
pictorial  explanation  of  the  procedures  which  can  be 
considered  for  testing  laboratory  assessment.) 

6.3.1.  On-Site  Assessment.  Before  initial 
accreditation  and  about  every  two  years  thereafter,  an 
on-site  assessment  of  each  laboratory  is  conducted  to 
determine  compliance  with  the  criteria.  Assessors  use 
standardized  checklists  so  each  laboratory  receives  a 
fair  assessment  in  relation  to  others?  however, 
assessors  have  considerable  latitude  in  judgments  about 
each  laboratory's  compliance  with  the  criteria 
depending  on  the  unique  circumstances  of  each 
laboratory.  The  assessors  are  selected  and  assigned  on 
the  basis  of  their  expertise  in  the  testing  techniques 
to  be  reviewed.  The  time  needed  to  conduct  an 
assessment  varies,  but  generally  one  week  or  less  is 
the  norm.  Every  effort  is  made  to  conduct  an 
assessment  with  as  little  disruption  as  possible  to  the 
normal  operations  of  the  laboratory.  The  assessors: 

6. 3. 1.1.  Meet  with  management  and  supervisory 
personnel  responsible  for  the  laboratory's 
activities  for  which  accreditation  is  being  sought 
to  acquaint  the  individuals  involved  and  to  set 
the  assessment  agenda. 
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6 .3. 1.2.  Examine  the  quality  system  employed  by 
the  laboratory,  its  major  equipment,  apparatus, 
and  facilities o 

6.3»1.3.  Assessors  thoroughly  review  the 
laboratory's  quality  manual  or  equivalent,  examine 
technician  notebooks  for  records  pertaining  to  the 
samples,  check  sample  identification  and  tracking 
procedures,  determine  whether  the  appropriate 
conditions  are  maintained,  and  examine  copies  of 
completed  test  reports. 

6. 3 .1.4.  Review  records  of  periodic  internal 
audits,  use  of  check  samples  or  participation  in 
round  robin  testing  or  other  similar  programs. 

6. 3. 1.5.  Review  representative  records  including 
competency  evaluations  for  all  staff  members  who 
perform  the  tests,  verification  records,  and 
sample  control  records. 

6. 3. 1.6.  Observe  demonstrations  of  testing 
techniques  and  discuss  them  with  the  technical 
personnel  to  assure  their  understanding  of  the 
procedures.  If  possible  the  history  of  one  or 
more  samples  from  receipt  to  final  issuance  of 
test  reports  is  traced. 

At  the  conclusion  of  the  assessment,  an  exit  briefing 
is  held  to  discuss  assessment  findings  with  laboratory 
management  and  identify  any  deficiencies  uncovered.  A 
written  summary  of  all  identified  deficiencies  is  left 
at  the  laboratory.  Assessment  forms  and  a written 
report  is  submitted  to  ICST  for  further  evaluation. 
The  laboratory  is  asked  to  respond  within  3 0 days  of 
the  date  of  the  exit  briefing  and  provide  documentation 
or  certification  that  the  specific  deficiencies  have 
been  corrected  or  that  specific  actions  are  being 
taken.  Any  laboratory  applying  for  initial 
accreditation  may  request  a delay  in  responding. 

If  any  deficiencies  are  noted  at  laboratories  which  are 
currently  accredited,  such  deficiencies  must  be 
corrected  within  30  days  after  the  exit  briefing  or  the 
laboratory  may  face  possible  suspension,  revocation  or 
expiration  of  its  accreditation.  When  test  systems  are 
identified  as  malfunctioning,  they  must  not  be  used 
until  corrective  action  has  been  completed.  Any 
deficiencies  noted  for  corrective  action  will  be 
subject  to  thorough  review  and  verification  during 
subsequent  assessments. 
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V/» 


ACCREDITATION  PROCESS 


Figure  1 


6.3.2.  Monitoring  Visits.  In  addition  to  regularly 
scheduled  assessments,  monitoring  visits  can  be  made  at 
any  time  during  the  accreditation  period.  Monitoring 
visits  serve  to  verify  reported  changes  in  the 
laboratory's  personnel,  facilities,  and  operations,  or 
to  explore  possible  reasons  for  poor  performance  in 
proficiency  testing.  The  scope  of  a monitoring  visit 
may  range  from  checking  a few  designated  items  to  a 
complete  review.  Failure  to  cooperate  with  assessors 
or  their  representatives,  may  be  grounds  for  adverse 
accreditation  action. 

6.3.3.  Proficiency  Testing.  Proficiency  testing  is  an 
integral  part  of  the  ICST  accreditation  process.  While 
the  existence  of  facilities,  equipment,  and  personnel 
which  satisfy  the  criteria  indicates  a laboratory's 
overall  capability  to  obtain  good  results,  an  analysis 
of  actual  test  results  for  certain  test  methods  is  also 
necessary  to  determine  if  the  overall  capability  does 
in  fact  produce  the  desired  results.  A laboratory's 
failure  to  participate  fully  in  the  conduct  of  required 
proficiency  testing  is  grounds  for  adverse 
accreditation  action. 

6.3.4.  Evaluation.  Evaluation  of  a laboratory  is 
conducted  by  ICST-approved  technical  experts  who  review 
records  on  the  applicant  laboratory  and  base  their 
decision  on: 

6. 3. 4.1.  information  provided  on  the  application; 


33 


6.3. 4.2.  on-site  assessment  reports? 

6. 3.4.3.  actions  taken  by  the  laboratory  to 
correct  deficiencies? 

6 c 3 . 4 . 4 . results  of  proficiency  testing?  and 

6. 3 .4. 5.  information  from  any  monitoring  visits 
performed  on  the  laboratory. 

If  the  technical  evaluation  reveals  additional 
deficiencies,  written  notification  describing  them  will 
be  made  to  the  laboratory.  The  laboratory  must  respond 
within  3 0 days  of  such  notification  and  provide 
documentation  or  certification  that  the  specified 
deficiencies  have  been  corrected.  Clarification  of 
some  issues  may  be  requested  by  telephone.  All 
deficiencies  must  be  corrected  before  accreditation  can 
be  granted  or  renewed. 

6. 3c 5.  Accreditation  Actions. 

The  Laboratory  Accreditation  Program  needs  to  address 
the  issues  of: 


6 0 3. 5.1.  Recommendation.  When  accreditation  is 
recommended,  the  recommendation  forms  the  basis 
for  granting  accreditation. 

6 .3. 5. 2.  Denial.  In  cases  where  denial  is 
recommended,  the  laboratory  is  notified  of  such 
action  and  the  reasons  for  denial  provided  in 
writing. 


6. 3. 5. 3.  Appeal.  When  denial  has  been  proposed, 
the  laboratory  may  request  a hearing,  under  United 
States  Code  (U.S.C.)  556. 

6. 3. 5. 4.  Renewal.  Accreditation  is  granted  for  a 
given  period  of  time  with  renewal  occurring  on  the 
same  anniversary  date. 

6. 3. 5. 5.  Termination.  A laboratory  may 
voluntarily  terminate  its  accreditation  by  written 
request  at  any  time. 

6. 3 .5.6.  Suspension.  If  an  accredited  laboratory 
develops  problems  or  deficiencies  which  are  of  a 
temporary  nature,  its  accreditation  may  be 
suspended  until  such  time  as  the  deficiencies  are 
resolved. 
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6. 3. 5. 7.  Revocation.  In  cases  where  a laboratory 
is  found  to  have  violated  the  terms  of  its 
accreditation,  the  accreditation  can  be  revoked. 
The  laboratory  may,  however,  be  given  the  option 
to  voluntarily  terminate  accreditation. 

6.3.6c  Public  Notification.  Accreditation  actions  and 
a directory  of  accredited  laboratories  is  published  on 
a routine  basis. 

6.4.  Responsibilities  of  Testing  Laboratories. 

6.4.1.  Carry  out  conformance  testing  and  provide  test 
reports  to  the  Client. 

6.4.2.  Fulfill  contractual  commitment  to  the 
Certification  Body.  * 

6.4.3.  Record  technical  test  details. 

6.4.4.  Communicate  the  approved  test  method  to  the 
Client  for  conformance  testing  preparation. 

6.4.5.  Treat  all  test  results  and  documents 
confidentially,  except  those  which  are  explicitly 
stated  as  public. 

6.4.6.  Comply  with  all  requirements  for  accreditation. 

6.4.7.  Comply  with  existing  laws.  Accreditation  does 
not  relieve  the  laboratory  of  the  need  to  observe  and 
comply  with  existing  Federal,  State,  and  local 
statutes,  ordinances,  or  regulations  that  may  be 
applicable  to  its  operation,  including  consumer 
protection  and  antitrust  laws. 

6.4.8.  Accredited  laboratories  are  encouraged,  within 
specified  limits,  to  publicize  their  accredited  status. 
The  major  restriction  is  that  advertising  must  not 
imply  product  certification  by  ICST  or  the  U.S. 
Government.  (Laboratories  and  their  Clients  may  not 
reference  their  accredited  status  in  consumer  media,  in 
product  advertising,  or  on  product  labels,  containers, 
or  packaging.) 

6.5.  Role  of  Certification  Body  in  Sponsoring  a Laboratory 
Accreditation  Program. 

6.5.1.  Ensure  the  testing  laboratory  is  accessible  to 
anyone,  including  foreign  Clients. 
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6.5.2.  Ensure  a model  contract  (e.gM  order  form)  is 
available  to  serve  as  agreement  between  Client  and 
testing  laboratory. 

6.5.3.  Ensure  that  accredited  laboratories  are 
available  to  meet  Client  needs.  Any  Client  requesting 
testing  for  a given  standard,  should  be  able  to  have 
his  product  tested  within  one  year  after  his  letter  of 
request  to  an  Accredited  Testing  Laboratory. 

6.5.4.  Establish  standard-specific  criteria  for 
assessing  competence  of  laboratories  to  perform 
testing.  (Information  contained  in  this  document  should 
assist  in  the  generic  aspects  of  assessment  plan 
development.)  On-site  assessment  can  be  carried  out  by 
the  Certification  Body  or  by  independent  assessors 
approved  by 'the  Certification  Body. 

6.5.5.  Construct  the  test  method  management  plan  with 
enough  flexibility  to  accommodate  technological 
developments . 

6.5.6.  Ensure  qualified  technical  experts,  assessors 
and  evaluators  are  used  for  accrediting  a testing 
laboratory. 

6.5.7.  Ensure  there  are  procedures  for  recommending 
the  laboratory  either  be  granted  or  denied 
accreditation.  This  recommendation  is  based  on  a 
review  of  the  evaluation  and  other  records  to  ensure 
that  all  technical,  financial,  and  administrative 
obligations  have  been  satisfied. 


7.  MUTUAL  RECOGNITION  OF  VALIDATION/ CERTIFICATION 

7.1.  Mutual  recognition  of  validation. 

Mutual  recognition  of  validation  is  issuing  a national 
certificate  solely  on  the  grounds  of  a test  report,  without 
testing  anew.  The  layout  of  both  test  reports  and 
certificates  should  be  harmonized  to  improve  mutuality  among 
testing  laboratories  and  certification  bodies,  and  to  ease 
translation. 

7.1.1.  Examine  bilateral  agreements  on  mutual 
recognition  of  certificates  which  exist  for  other 
conformance  testing  programs,  e.g.,  language  compilers. 

7.2.  Common  Recognition. 

To  help  achieve  common  recognition  (at  both  the  national  and 
international  level)  for  test  methods,  interface  is 
encouraged  through: 
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7.2.1.  The  certification  system  to  provide  the  right 
for  every  testing  laboratory  to  attend  the  testing  by 
any  other  laboratory  once  a year  in  order  to  allow 
reciprocal  observation  for  the  sake  of  identical 
results. 


7.2.2.  Any  Client  worldwide,  should  be  able  to  address 
the  Certification  Body. 

7.3.  Considerations  and  Issues. 

Considerations  and  issues  need  to  be  resolved  in  order  to 
achieve  common  recognition  of  validation  test  reports  for 
products  offered  by  multi-national  companies  who  trade  with 
other  countries. 
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TERMS  AND  DEFINITIONS 


ACCEPTANCE  TESTING.  Formal  testing  conducted  to  determine  whether 
a software  system  satisfies  its  acceptance  criteria  and  to  enable 
the  customer  to  determine  whether  to  accept  the  system.  Formal 
testing  includes  the  planning  and  execution  of  several  kinds  of 
tests,  (e.g,f  functional,  volume,  performance  tests)  to 
demonstrate  the  implemented  software  satisfies  the  customer 
requirements  for  the  software  system. 

ACCREDITATION.  The  formalized  initial  and  continuing  acceptance 
of  a testing  laboratory  as  competent  to  carry  out  specific  tests. 

APPROVED  TEST  METHOD.  An  organized  system  under  which,  on  a 
uniform  and  equitable  basis,  similar  products  or  services  may  be 
certified  to  specified  standards. 

ASSESSORS.  Selected  to  conduct  an  on-site  assessment  of  a 
particular  laboratory  on  the  basis  of  how  well  their  individual 
experience  matches  the  type  of  testing  to  be  assessed.  The 
laboratory  has  the  right  to  appeal  the  assignment  of  an  assessor 
and  may  request  an  alternate. 

CERTIFICATE  OF  CONFORMANCE.  A document  attesting  that  a product 
or  a service  is  in  conformance  with  specific  standards  or 
technical  specifications  as  determined  through  use  of  a specified 
test  method. 

CERTIFICATION.  The  procedure  by  which  a product (s)  or  service (s) 
becomes  certified  after  passing  tests  for  conformance  to  a 
standard. 

CERTIFICATION  BODY.  An  impartial  body,  governmental  or  non- 
governmental, possessing  the  necessary  competence  and  reliability 
to  operate  or  accredit  operation  of  a certification  system,  and 
in  which  the  interests  of  all  parties  concerned  with  the  function 
of  the  system  are  represented. 

CERTIFICATION  MARK.  The  Certification  Body's  validating  sign, 
symbol,  or  letter  that  identifies  a product (s)  or  service (s)  as 
being  certified. 

CERTIFICATION  SYSTEM.  A system  having  its  own  rules  of  procedure 
and  management,  for  carrying  out  conformance  certification. 

CERTIFIED.  Attested  by  the  manufacturer/vendor  under  the 
procedures  of  an  accredited  testing  laboratory  as  satisfying  the 
requirements  of  the  referenced  standard (s) . 

CLIENT.  As  used  in  this  Plan,  Client  refers  to  any  organization 
or  person  who  employs  a testing  laboratory  for  any  purpose. 
Thus,  "Client"  can  refer  to  a commercial  organization  or 
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Certification  Body  who  uses  the  services  of  the  testing 
laboratory.  A Client  is  someone  who  wants  products  to  be  tested. 

CONFORMANCE  CERTIFICATION.  The  action  of  certifying  by  means  of  a 
certificate  of  conformance  or  mark  of  conformance  that  a product 
or  service  is  in  conformance  with  specific  standards  or  technical 
specifications . 

CONFORMANCE  SUMMARY  REPORT.  A document  written  after  conformance 
testing  which  records  details  about  the  IUT  and  the  environment 
in  which  it  was  tested.  The  overall  summary  for  the  test  cases 
executed,  and,  for  each  test  case,  the  test  purpose,  analysis, 
and  verdict. 

CONFORMANCE  TESTING.  The  testing  of  a candidate  product  for  the 
existence  of  specific  characteristics  required  by  a standard? 
testing  the  extent  to  which  an  IUT  is  a conforming 
implementation. 

CONFORMANCE  WITH  STANDARDS  OR  TECHNICAL  SPECIFICATIONS.  The 
conformance  of  a product  or  a service  with  all  the  requirements 
of  specific  standards  or  technical  specifications. 

CONFORMING  IMPLEMENTATION.  An  implementation  under  test  which  is 
shown  to  satisfy  the  conformance  requirements,  consistent  with 
the  capabilities  and  options  stated  in  the  test  suite  pass/fail 
documentation . 

CONTROL  BOARD.  An  impartial  body,  governmental  or  non- 
governmental,  possessing  the  necessary  competence  and  reliability 
to  approve  and  maintain  proficient  test  suites  for  the  testing 
program,  and  in  which  the  interests  of  all  parties  concerned  with 
the  function  of  the  tests  are  represented. 

ERROR.  A failure  of  a IUT  to  meet  its  test  specifications. 

EVALUATORS.  Selected  to  provide  a second  opinion,  if  necessary, 
and  to  review  the  record  including  the  application,  assessment 
report,  deficiencies,  corrections  to  deficiencies,  and 
proficiency  test  results  and,  based  on  this  record,  to  recommend 
whether  accreditation  should  be  granted. 

"FAIL”  VERDICT.  A verdict  given  when  the  observed  outcome 
matches  an  outcome  identified  or  categorized  as  "fail"  in  the 
test  case  specification. 

FALSIFICATION  TESTING.  Method  using  sample  cases  that  test  as 
many  of  the  requirements  of  the  standard  as  are  feasible.  The 
test  suite  tries  to  find  errors  in  the  implementation.  If  errors 
are  found,  one  can  correctly  deduce  the  implementation  does  not 
conform  to  the  standard;  however,  the  absence  of  errors  does  not 
necessarily  imply  the  converse.  The  absence  of  errors  implies 
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either  that  the  implementation  conforms  to  the  standard  o£  that 
the  test  suite  was  not  comprehensive  enough  to  find  errors. 
Falsification  testing  can  only  prove  nonconformance.  (Proofs  of 
Correctness  Tests  prove  conformance  to  the  standard.) 

FIELD  TESTING.  Examining  the  test  method  and  test  suite  for  the 
quality  necessary  to  perform  a successful  conformance  test  to  the 
standard. 

IMPLEMENTATION  UNDER  TEST  (IUT)  . That  part  of  a standard  which 
is  to  be  studied  under  testing,  which  should  be  an  implementation 
of  one  or  more  characteristics  of  the  standard. 

"INCONCLUSIVE"  VERDICT.  A verdict  given  when  the  observed 
outcome  either  matches  an  outcome  identified  or  categorized  as 
" inconclusive"  in  the  test  case  specification  or  does  not  match 
any  identified  outcome  in  the  test  case  specification. 

IUT.  see  Implementation  Under  Test. 

LABORATORY  ACCREDITATION.  A formal  recognition  that  a testing 
laboratory  is  competent  to  carry  out  specific  tests  or  specific 
types  of  tests. 

MANUFACTURER'S  DECLARATION  OF  CONFORMANCE.  The  action  by  which  a 
manufacturer  declares  under  his  sole  responsibility,  by  means  of 
a 'declaration  of  conformance,'  that  the  product  is  in 
conformance  with  designated  standards  or  other  technical 
specifications,  without  being  under  the  procedures  of  a third- 
party  certification  system.  Note:  this  term  is  preferred  over 
self-certification  to  avoid  confusion  with  the  certification 
process  in  general. 

NONCONFORMANCE.  When  a product  which  has  undergone  conformance 
testing  at  an  accredited  laboratory  does  not  pass  the  designated 
number  of  tests  to  qualify  for  conformance  to  the  standard. 

"PASS"  VERDICT.  A verdict  given  when  the  observed  outcome 
matches  an  outcome  identified  as  "pass"  in  the  test  case 
specification. 

PROFICIENCY  TESTING  (LABORATORY) . Determination  of  laboratory 
testing  performance  by  means  of  interlaboratory  test  comparisons. 

PROOF  OF  CORRECTNESS.  The  use  of  techniques  of  mathematical  logic 
to  infer  that  a relation  between  program  variables  assumed  true 
at  program  entry  implies  that  another  relation  between  program 
variables  holds  at  program  exit. 

REFERENCE  MATERIAL.  A material  or  substance  one  or  more 
properties  of  which  are  sufficiently  well  established  to  be  used 
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for  the  calibration  of  an  apparatus,  the  assessment  of  a 
measurement  method,  or  for  assigning  values  to  materials. 

SEMANTICS.  The  functional  description,  it  defines  precisely  what 
must  be  done,  but  not  how  it  is  to  be  done  (as  does  the  syntax)  . 
Generally  specified  in  narrative  form  using  the  English  language. 

SYNTAX.  Can  consist  of  verbs  in  a programming  language  to  access 
the  function  or,  in  the  case  of  graphics  standards,  the 
"bindings"  to  existing  programming  languages  to  access  the 
graphics  functions  in  the  most  natural  way  for  programmers, 
depending  on  the  language  they  are  using. 

SYSTEM  UNDER  TEST  (SUT) . The  computer  hardware  and/or  software 
in  which  the  IUT  resides. 

SUT.  see  System  Under  Test. 

TECHNICAL  EXPERTS.  Respected  peers  in  their  field  used  as 
assessors  and  evaluators,  and  selected  through  evaluation  of 
their  professional/academic  achievements,  experience  in  the  field 
of  testing,  management  awareness,  potential  for  conflict-of- 
interest,  and  tact  in  dealing  with  people. 

TEST  METHOD.  A specified  technical  procedure  to  determine  one  or 
more  specified  characteristics  of  a material  or  product.  Note  1: 
test  method  includes  the  test  software.  Note  2:  part  of  such  a 
technical  procedure  may  be  described  by  statements  in  some 
programming  language.  The  relevant  software  package  and  any 
hardware  needed  (including  reference  material)  is  understood  to 
be  part  of  the  test  method. 

TEST  REPORT.  A document  which  presents  the  test  results  and  other 
information  relevant  to  the  test. 

TEST  SUITE.  A complete  set  of  tests  necessary  to  perform 
conformance  testing  for  an  IUT,  together  with  the  information  for 
determining  the  order  in  which . they  are  to  be  executed  when  in 
their  executable  form. 

TESTING  LABORATORY.  A laboratory  which  measures,  examines, 
tests,  calibrates,  or  otherwise  determines  the  characteristics  or 
performance  of  materials  or  products.  Note:  "Testing  Laboratory" 
in  the  context  of  this  document  may  mean  (1)  a body  corporate, 
(2)  one  of  its  subdivisions,  (3)  the  laboratory  proper  (office 
and  equipment) , or  (4)  the  testing  service  functions  for  a 
specific  standard. 

VALIDATION.  Determination  of  the  correctness  of  the  program  or 
software  produced  to  support  a specific  standard  or  set  of 
standards . 
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If 


VERDICT . Statement  of  "pass,"  "fail,"  or  "inconclusive” 
concerning  conformance  of  an  IUT  and  made  with  respect  to  a test 
case  that  has  been  executed. 


VERIFICATION.  The  demonstration  of  consistency,  completeness,  and 
correctness  of  the  product  software. 


46 


SAMPLE  APPLICATION  FOR  TESTING  LABORATORY 


Appendix  B 


47 


SAMPLE  APPLICATION  FOR  TESTING  LABORATORY 

i 

1.  Applicant's  name  and  address s 

Telephone  No. s 

Telex  No. : 

2 o Test  Laboratory3  name  and  address  (if  different  from 

paragraph  1)2 

Telephone  No. : 

Telex  No. s 

3 o Senior  management 

3=1.  Names  and  titles  of  the  senior  executives  of  the 
Test  Laboratory1  and  of  the  testing  laboratories 
for  which  recognition  is  being  sought: 

3.2.  Name  and  title  of  the  person  responsible  for  the 
Quality  Management  System  in  the  testing 
laboratory: 

3.3.  Name  and  title  of  the  principal  contact  nominated 
by  the  testing  laboratory,  and  of  his  deputy: 

3.4.  Operating  departments  of  the  testing  laboratory 
for  which  recognition  is  being  sought.  (Show  on 
a separate  sheet  to  be  attached  either  as  a list 
or  as  an  organization  chart  of  the  Test 
Laboratory1 . ) : 

4 . Employees 

4  o 1 . Total  number  in  Test  Laboratory1: 

4.2.  Total  number  in  testing  laboratory  for  which 
recognition  is  being  sought: 

5 . Equipment 

List  on  a separate  sheet  the  major  items  of  test  equipment 

available  for  use  in  the  area  for  which  recognition  is  being 

sought. 

6.  Test  facilities  and  services 

6.1.  List  on  a separate  sheet  the  testing  services  for 
which  recognition  is  being  sought,  indicating  for 


3 "Test  Laboratory"  refers  here  to  the  corporate  entity 
having  final  authority  over  the  "testing  laboratory"  seeking 
recognition.  In  some  cases  this  may  mean  one  of  the 
corporate's  subdivisions  or  the  laboratory  proper  (office  and 
equipment) . 
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each  service  any  limits  between  which  it  will 
operate,  and  the  published  specifications  against 
which  the  testing  will  be  performed. 

6.2.  If  recognition  by  other  bodies  or  authorities  are 
held  in  the  area  for  which  recognition  is  being 
sought,  please  give  details,  e.g.,  subcontractor. 

6.3.  What  type  of  testing  is  to  be  subcontracted  in 
respect  of  the  recognition  being  sought? 

7 . Other  information 

7.1.  Document,  where  applicable,  how  the  testing 
laboratory  may  be  related  to  external 
organizations  or  to  components  within  its  own 
parent  organization. 

7.2.  Give  any  other  information  which  you  consider 

could  be  of  assistance  to  the  assessment  team  (on 
a separate  sheet  if  necessary) . 

8.  Quality  Management  Policy 

8.1.  Are  policy  and  procedures  for  the  operation  of 
the  testing  laboratory  contained  in  a document 
such  as  a Quality  Manual? 

8.2.  Has  the  person  responsible  for  guality  management 
the  responsibility  and  authority  to  identify 
quality  problems  and  initiate  effective 
solutions? 

8.3.  Does  the  Quality  Manual  contain  procedures  for 

the  supervision  of  any  unqualified  staff? 

8.4.  Is  there  a prescribed  audit  procedure  for 

checking  quality  management  functions? 

9 . Work  instructions 

9.1.  Are  manuals,  work  instructions,  and  regulations 
to  be  used  by  staff  readily  available? 

9.2.  Is  there  a system  for  updating,  implementing,  and 
recording  changes  to  these  documents? 

9.3.  Are  documents  available  for  each  testing 

operation? 

9.4.  Are  documents  and  reference  data  maintained  in  an 
up-to-date  condition? 

9.5.  Is  obsolete  data  promptly  removed  from  documents, 
etc.  ? 
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10.  Personnel 

10.1.  Have  standards  of  professional  ability,  skills, 
and  job  descriptions  been  prescribed  where 
necessary? 

10.2.  Are  training  methods  applied  to  attain  and 
maintain  skills  with  due  attention  to  quality 
requirements? 

11.  Test  equipment 

11.1.  Does  the  Quality  Management  System  specify  that 
the  equipment  is  of  an  accuracy  compatible  with 
the  testing  undertaken? 

11.2.  Is  a record  maintained  of  all  test  equipment? 

12 . Test  procedures 

12.1.  Are  test  methods  and  procedures  recorded  which 
are  not  called  up  in  specifications,  manuals, 
etc.  ? 

12.2.  Are  the  environments  in  which  tests  are  conducted 
and  results  recorded  suitable  to  ensure  their 
accuracy? 

12 o 3 o Do  environmental  test  facilities  exist? 

12.4.  Is  there  control  of  access  to  the  testing  areas? 

12.5.  Is  there  a prescribed  system  for  detecting 
deficiencies  in  testing  and  their  causes,  and  for 
correcting  unfavorable  trends? 

13 . Handling  and  storage 

13.1.  Are  work  and  inspection  instructions  prescribed 
and  implemented  for  the  handling,  storage,  and 
return  to  the  submitter  of  software  and 
equipment? 

13.2.  Are  appropriate  storage  areas  arranged  to  prevent 
deterioration  or  damage  to  the  products 
concerned? 

13.3.  Are  storage  areas  accessible  only  to  authorized 
persons? 


14  o Records 

14.1.  Is  there  a prescribed  system  for  recording  the 
method  and  results  of  testing  activities? 
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14.2.  Are  observations  and  calculations  recorded  and 
stored  as  to  provide  a permanent  test  record? 

14.3.  Are  there  arrangements  for  ensuring  that  records 
are  current,  complete,  accurate,  and  held 
confidential  where  required? 


15 . Test  reports 

15.1.  Do  test  reports  contain  all  the  information 
required  for  such  by  ICST? 

15.2.  Is  the  testing  laboratory  prepared  to  make 
arrangements  to  send  copies  of  test  reports  to 
ICST,  where  required,  on  a strictly  confidential 
basis? 


IS.  Preparedness  for  assessment 

16.1.  Are  you  satisfied  that  you  can  meet  all  the 
requirements  prescribed  herein? 

16.2.  At  what  date  will  the  testing  laboratory  be  ready 
for  assessment? 

16.3.  Is  there  any  special  urgency  for  assessment? 

If  so  what  is  the  reason? 


Applicant ' s name 


Signature  and  title  of  person  authorized  to  sign  for  the 

Applicant 

( S ignature/Title ) 


Date 
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APPLICATION  FOR  ASSESSMENT  AS  A RECOGNIZED 
TESTING  LABORATORY 

Name  of  applicant:  Name  and  address  of  testing 

laboratory  if  different: 


Address: 


Name  of  Contact: 
Title: 


Name  of  Contact: 
Title: 


Test  or  series  of  tests  for  which  recognition  is  sought: 


The  testing  laboratory  hereby  agrees  to: 

a)  conform  to  the  requirements  for  a recognized  testing  laboratory; 

b)  pay  any  costs  connected  with  assessment  and  administration  in  the 
pre-recognition  stage  irrespective  of  the  eventual  granting  of 
recognition. 


Signed: 


(Title) 


Date : 


Figure  1 
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MODEL  CERTIFICATE  OF  ACCREDITATION 


APPENDIX  C 
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MODEL  CERTIFICATE  OF  ACCREDITATION 


This  is  to  certify  the  laboratory  named  herein  has  been  accredited 
using  the  latest  official  version,  version  number,  of  appropriate 
standard  test  method ; therefore  the  laboratory  conforms  to  the 
accreditation  criteria  until  the  expiration  date  shown  below  on  this 
certificate.  This  laboratory  must  be  recertified  at  least  once  every 
two  years,  using  this  certificate  date  as  the  anniversary  date  for  re- 
accreditation. Accreditation  must  occur  at  least  30  days  prior  to  the 
expiration  date  of  this  Certificate  or  a new  Certificate  must  be  issued 
by  the  National  Bureau  of  Standards,  Institute  for  Computer  Sciences 
and  Technology  by  the  expiration  date  in  order  to  keep  the 
accreditation  of  the  testing  laboratory  in  good  standing  and  eligible 
for  inclusion  in  the  Accreditation  Directory  List  for  the  following 
year. 

Laboratory  Name  

Laboratory  Address  


Point  of  Contact:  Name/Phone 


Accreditation  for  Standard:  Title/Number 


Test  Method  Version  Number  

Accreditation  Assessment  Report 
EXPIRATION  DATE 


ICST  Director  Signature 
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